Single Image Super-resolution Problem Research Articles

GSDD: Generative Space Dataset Distillation for Image Super-resolution

Single image super-resolution (SISR), especially in the real world, usually builds a large amount of LR-HR image pairs to learn representations that contain rich textural and structural information. However, relying on massive data for model training not only reduces training efficiency, but also causes heavy data storage burdens. In this paper, we attempt a pioneering study on dataset distillation (DD) for SISR problems to explore how data could be slimmed and compressed for the task. Unlike previous coreset selection methods which select a few typical examples directly from the original data, we remove the limitation that the selected data cannot be further edited, and propose to synthesize and optimize samples to preserve more task-useful representations. Concretely, by utilizing pre-trained GANs as a suitable approximation of realistic data distribution, we propose GSDD, which distills data in a latent generative space based on GAN-inversion techniques. By optimizing them to match with the practical data distribution in an informative feature space, the distilled data could then be synthesized. Experimental results demonstrate that when trained with our distilled data, GSDD can achieve comparable performance to the state-of-the-art (SOTA) SISR algorithms, while a nearly ×8 increase in training efficiency and a saving of almost 93.2% data storage space can be realized. Further experiments on challenging real-world data also demonstrate the promising generalization ability of GSDD.

Advanced Reference-Constrained Image Restoration Algorithm

Image restoration is always a long-term problem in low-level computer vision, which has many practical applications. In this paper, we devote to reconstructing a single image from compressive measurements with the assistance of one or multiple reference images, by using the similarity prior information of them. More specifically, we propose to bring in the concept of Regularization by Denoising, which leverages existing denoising engine to regularize inverse problems, and illustrate how the reference images and this powerful algorithm can be merged into a new highly effective recovery approach, which is named Reference Image Constrained Regularization by Denoising for solving various image restoration tasks. Since the similarity between the reference and the ground truth is not guaranteed, or the degree of the similarity is different, we extend the proposed algorithm by adding adaptive weighting matrix to the difference between the target image and the reference images to further improve the performance. We evaluate experimentally the proposed algorithm in the image deblurring and single image super-resolution problems. Furthermore, we also carry out our method on video super-resolution task, in the case of multiple reference images available. We compare the advanced reference-constrained image restoration algorithm with previous methods about reconstruction accuracy and computation cost. The experimental results show that the proposed method outperforms state-of-the-art algorithms, including Regularization by Denoising, in terms of both the Peak Signal to Noise Ratio and the visual performance.

SRResNet Performance Enhancement Using Patch Inputs and Partial Convolution-Based Padding

Due to highly underdetermined nature of Single Image Super-Resolution (SISR) problem, deep learning neural networks are required to be more deeper to solve the problem effectively. One of deep neural networks successful in the Super-Resolution (SR) problem is ResNet which can render the capability of deeper networks with the help of skip connections. However, zero padding (ZP) scheme in the network restricts benefits of skip connections in SRResNet and its performance as the ratio of the number of pure input data to that of zero padded data increases. In this paper. we consider the ResNet with Partial Convolution based Padding (PCP) instead of ZP to solve SR problem. Since training of deep neural networks using patch images is advantageous in many aspects such as the number of training image data and network complexities, patch image based SR performance is compared with single full image based one. The experimental results show that patch based SRResNet SR results are better than single full image based ones and the performance of deep SRResNet with PCP is better than the one with ZP.

Single image super-resolution using Wasserstein generative adversarial network with gradient penalty

Due to its strong sample generating ability, Generative Adversarial Network (GAN) has been used to solve single image super-resolution (SISR) problem and obtains high perceptual quality super-resolution (SR) images. However, GAN suffers from the disadvantage of training instability, even fails to converge. In this paper, a new SISR method is proposed based on Wasserstein GAN, which is a training more stable GAN with Wasserstein metric. To further increase the SR performance and make the training process more easier and stable, two modifications are made on the original WGAN. First, a gradient penalty (GP) is adopted to replace weight clipping. Second, a new residual block with “pre-activation” of the weight layer is constructed in the generators of WGAN. Extensive experiments show that the proposed method yields superior SR performance than original GAN based SR methods and many other methods in accuracy and perceptual quality of ×4 magnification factor on four diverse testing datasets.

ADMM-Based Residual Whiteness Principle for Automatic Parameter Selection in Single Image Super-Resolution Problems

We propose an automatic parameter selection strategy for the single image super-resolution problem for images corrupted by blur and additive white Gaussian noise with unknown standard deviation. The proposed approach exploits the structure of both the down-sampling and the blur operators in the frequency domain and computes the optimal regularisation parameter as the one optimising a suitably defined residual whiteness measure. Computationally, the proposed strategy relies on the fast solution of generalised Tikhonov ell _2–ell _2 problems as proposed in Zhao et al. (IEEE Trans Image Process 25:3683–3697, 2016). These problems naturally appear as substeps of the Alternating Direction Method of Multipliers used to solve single image super-resolution problems with non-quadratic, non-smooth, sparsity-promoting regularisers both in convex and in non-convex regimes. After detailing the theoretical properties allowing to express the whiteness functional in a compact way, we report an exhaustive list of numerical experiments proving the effectiveness of the proposed approach for different type of problems, in comparison with well-known parameter selection strategies such as, e.g., the discrepancy principle.

Best-Buddy GANs for Highly Detailed Image Super-resolution

We consider the single image super-resolution (SISR) problem, where a high-resolution (HR) image is generated based on a low-resolution (LR) input. Recently, generative adversarial networks (GANs) become popular to hallucinate details. Most methods along this line rely on a predefined single-LR-single-HR mapping, which is not flexible enough for the ill-posed SISR task. Also, GAN-generated fake details may often undermine the realism of the whole image. We address these issues by proposing best-buddy GANs (Beby-GAN) for rich-detail SISR. Relaxing the rigid one-to-one constraint, we allow the estimated patches to dynamically seek trustworthy surrogates of supervision during training, which is beneficial to producing more reasonable details. Besides, we propose a region-aware adversarial learning strategy that directs our model to focus on generating details for textured areas adaptively. Extensive experiments justify the effectiveness of our method. An ultra-high-resolution 4K dataset is also constructed to facilitate future super-resolution research.

Learn to Zoom in Single Image Super-Resolution

In this letter, we propose a novel solution to the problem of single image super-resolution at multiple scaling factors, including the extreme case of 8x, with a single network architecture. In applications where only a detail needs to be super-resolved, traditional solutions must choose to use as input either the low-resolution detail, thus losing the information about the context, or the whole low-resolution image and then crop the desired output detail, which is quite wasteful in terms of computations and storage. To address both of these issues we propose ZoomGAN, a model that takes as input the whole low-resolution image, which we call context, and a binary mask that specifies with a box which image detail in the low-resolution image to magnify. The output of ZoomGAN has the same size as the inputs so that the scaling factor is implicitly defined by the arbitrary size of the mask box. To encourage a realistic and high-quality output, we combine adversarial training with a perceptual loss. We use two discriminators: one promotes the similarity between the distributions of real and generated details and the other promotes the similarity between the distributions of real and generated (detail, context) pairs. We evaluate ZoomGAN with several experiments on several datasets and show that it achieves state of the art performance on zoomed in details in terms of the LPIPS and PI perceptual metrics, while being on par in terms of the PSNR distortion metric.

Learning Curvelet-based Directional Dictionaries for Single Image Super Resolution

Learning and reconstruction-based methods are the two main approaches to the solve single image super resolution (SISR) problem. In this paper, to exploit the advantages of both learning based and reconstruction based approaches, we propose a new SISR framework by combining them, which can effectively utilize their benefits. The external directional dictionaries (EDD) are learned from external high quality images. Additionally, we embeded the nonlocal means (NLM) filter and an isotropic total variation (TV) scheme in the reconstruction based method. We suggest a new supervised clustering scheme via curvelet based direction extraction method (CCDE) to learn the external directional dictionaries from candidate patches with sharp edges. Each input patch is coded by all the EDD. Each of the reconstructed patches under different EDD is applied with a weighted penalty to characterize the given input patch. To disclose new details, the local smoothness and nonlocal self-similarity priors are added on the recovered patch by TV scheme and NLM filter. Extensive experimental results validate the effectiveness and robustness of the proposed method comparing with the state-of- the-art algorithms in SISR methods. Our proposed schemes can retrieve more fine structures and obtain superior results than the competing methods with the scaling factors of 2 and 3.

Image Super-Resolution Through Compressive Sensing-based Recovery.

The primary aim of image super-resolution techniques is to produce a high resolution (HR) image from a low resolution (LR) image efficiently. Deep learning algorithms are being extensively used to address the ill-posed problem of single image super-resolution which requires extremely large data-sets and high processing power. When one does not have access to large data-sets or have limited processing power, an alternative technique may be in order. In this study, we have developed a novel positive scale image resizing method inspired by compressive sensing (CS). We have considered the image super-resolution as a CS recovery problem in which a low resolution image is assumed as a compressed measurement and the required interpolated image is treated as output of the CS-based recovery. In the proposed HR recovery method, a deterministic binary block diagonal measurement matrix, (DBBD), is used as measurement matrix since it maintains the visual similarity between the low and high resolution images. Then along with a sparsification matrix, the sparse representation of HR image is first recovered and subsequently the dense HR image is obtained. The proposed method is applied to medical and non-medical images. The HR images obtained using the traditional proximal, bilinear and bi-cubic interpolation techniques are compared with those obtained using the proposed method. The proposed CS inspired method delivers superior HR images than the traditional techniques. The superiority of the proposed method is attributed to the unique usage of the DBBD matrix and the CS recovery algorithm to obtain a high resolution image without any prior training data-set.

Balanced distortion and perception in single-image super-resolution based on optimal transport in wavelet domain

Single image super-resolution (SISR) is a classic ill-posed problem in computer vision. In recent years, deep-learning-based (DL-based) models have achieved promising results with the SISR problem. However, most existing methods suffer from an intrinsic trade-off between distortion and perceptual quality. To satisfy the requirements in different real-world situations, the balance of distortion and visual quality for image super-resolution is a critical issue. In DL-based models, the uses of hybrid loss (i.e., the combination of the distortion loss and the perceptual loss) and network interpolation are two common approaches to balancing the distortion and perceptual quality of super-resolved images. However, these two kinds of methods lack flexibility and hold strict constraints on network architectures. In this paper, we propose an image-fusion interpolation method for image super-resolution, which can balance the distortion and visual quality of super-resolved images, based on the optimal transport theory in the wavelet domain. The advantage of our proposed method is that it can be applied to any pretrained DL-based model, without any requirement from the network architecture and parameters. In addition, our proposed method is parameter-free and can run fast without using a GPU. Compared with existing state-of-the-art SISR methods, experiment results show that our proposed method can achieve a better balance between the distortion and visual quality in super-resolved images.

Adversarial Multi-Path Residual Network for Image Super-Resolution.

Recently, deep convolutional neural networks have demonstrated remarkable progresses on single image super-resolution (SR) problem. However, most of them use more deeper and wider networks to improve SR performance, which is not practical in real-world applications due to large complexity, high computation cost, and low efficiency. In addition, they cannot provide high perception quality and guarantee objective quality simultaneously. To address these limitations, we in this paper propose a novel Adversarial Multipath Residual Network (AMPRN), which can largely suppress the number of network parameters and achieve a higher SR performance compared with the state-of-the-art methods. More specifically, we propose a multi-path residual block (MPRB) for multi-path residual network (MPRN) with fewer network parameters, which can extract abundant local features by fully using features from different paths generated by channel slices. These hierarchical features from all the MPRBs are then jointly aggregated by global gradual feature fusion. Following MPRN, we construct an adversarial gradient network with a gradient loss to make the gradient distribution of the generated SR images and ground truth image closer. In this way, the generated SR images of our model can provide high perception quality and objective quality. Finally, several experimental results demonstrate that our AMPRN achieves better performance in comparison with fewer parameters than the state-of-the-art methods.

Deconvolutional neural network for image super-resolution

This study builds a fully deconvolutional neural network (FDNN) and addresses the problem of single image super-resolution (SISR) by using the FDNN. Although SISR using deep neural networks has been a major research focus, the problem of reconstructing a high resolution (HR) image with an FDNN has received little attention. A few recent approaches toward SISR are to embed deconvolution operations into multilayer feedforward neural networks. This paper constructs a deep FDNN for SISR that possesses two remarkable advantages compared to existing SISR approaches. The first improves the network performance without increasing the depth of the network or embedding complex structures. The second replaces all convolution operations with deconvolution operations to implement an effective reconstruction. That is, the proposed FDNN only contains deconvolution layers and learns an end-to-end mapping from low resolution (LR) to HR images. Furthermore, to avoid the oversmoothness of the mean squared error loss, the trained image is treated as a probability distribution, and the Kullback–Leibler divergence is introduced into the final loss function to achieve enhanced recovery. Although the proposed FDNN only has 10 layers, it is successfully evaluated through extensive experiments. Compared with other state-of-the-art methods and deep convolution neural networks with 20 or 30 layers, the proposed FDNN achieves better performance for SISR.

Fast multi‐spectral image super‐resolution via sparse representation

Sparse reconstruction is used to solve the inverse problem of single image super-resolution (SR) as a patch-based sparsity promoting regularisation problem. A coupled trained overcomplete dictionary from high-resolution (HR) and low-resolution (LR) image patches containing significant features is proposed using sparse representation to produce HR patches from their LR counterparts. In this study, the authors develop a multi-core single image SR technique for LR multi-spectral images based on patch-wise sparse representation coupled with morphological component analysis driven feature extraction. Simulations are carried out to evaluate the proposed method using real remote sensing images of a few Indian satellites, RESOURCESAT-2 and CARTOSAT-2, as well as other satellites, such as QuickBird etc. Results are also compared with other existing SR methods to establish the superiority of the proposed method in terms of both objective metrics and visual analysis.

Adaptive Residual Channel Attention Network for Single Image Super-Resolution

Single image super-resolution (SISR) is a traditional image restoration problem. Given an image with low resolution (LR), the task of SISR is to find the homologous high-resolution (HR) image. As an ill-posed problem, there are works for SISR problem from different points of view. Recently, deep learning has shown its amazing performance in different image processing tasks. There are works for image super-resolution based on convolutional neural network (CNN). In this paper, we propose an adaptive residual channel attention network for image super-resolution. We first analyze the limitation of residual connection structure and propose an adaptive design for suitable feature fusion. Besides the adaptive connection, channel attention is proposed to adjust the importance distribution among different channels. A novel adaptive residual channel attention block (ARCB) is proposed in this paper with channel attention and adaptive connection. Then, a simple but effective upscale block design is proposed for different scales. We build our adaptive residual channel attention network (ARCN) with proposed ARCBs and upscale block. Experimental results show that our network could not only achieve better PSNR/SSIM performances on several testing benchmarks but also recover structural textures more effectively.

Multi-Image Super Resolution of Remotely Sensed Images Using Residual Attention Deep Neural Networks

Convolutional Neural Networks (CNNs) consistently proved state-of-the-art results in image Super-resolution (SR), representing an exceptional opportunity for the remote sensing field to extract further information and knowledge from captured data. However, most of the works published in the literature focused on the Single-image Super-resolution problem so far. At present, satellite-based remote sensing platforms offer huge data availability with high temporal resolution and low spatial resolution. In this context, the presented research proposes a novel residual attention model (RAMS) that efficiently tackles the Multi-image Super-resolution task, simultaneously exploiting spatial and temporal correlations to combine multiple images. We introduce the mechanism of visual feature attention with 3D convolutions in order to obtain an aware data fusion and information extraction of the multiple low-resolution images, transcending limitations of the local region of convolutional operations. Moreover, having multiple inputs with the same scene, our representation learning network makes extensive use of nestled residual connections to let flow redundant low-frequency signals and focus the computation on more important high-frequency components. Extensive experimentation and evaluations against other available solutions, either for Single or Multi-image Super-resolution, demonstrated that the proposed deep learning-based solution can be considered state-of-the-art for Multi-image Super-resolution for remote sensing applications.

Suitability of Single Image Super Resolution Models for Video Super Resolution

This project is an attempt to understand the suitability of the Single image super resolution models to video super resolution. Super Resolution refers to the process of enhancing the quality of low resolution images and video. Single image super resolution algorithms refer to those algorithms that can be applied on a single image to enhance its resolution. Whereas, video super resolution algorithms are those algorithms that are applied to a sequence of frames/images that constitute a video to enhance its resolution. In this paper we determine whether single image super resolution models can be applied to videos as well. When images are simply resized in Open CV, the traditional methods such as Interpolation are used which approximate the values of new pixels based on nearby pixel values which leave much to be desired in terms of visual quality, as the details (e.g. sharp edges) are often not preserved. We use deep learning techniques such as GANs (Generative Adversarial Networks) to train a model to output high resolution images from low resolution images. In this paper we analyse suitability of SRGAN and EDSR network architectures which are widely used and are popular for single image super resolution problem. We quantify the performance of these models, provide a method to evaluate and compare the models. We further draw a conclusion on the suitability and extent to which these models may be used for video super resolution. If found suitable this can have huge impact including but not limited to video compression, embedded models in end devices to enhance video output quality.

Deep Learning-Based Single Image Super-Resolution: An Investigation for Dense Scene Reconstruction with UAS Photogrammetry

The deep convolutional neural network (DCNN) has recently been applied to the highly challenging and ill-posed problem of single image super-resolution (SISR), which aims to predict high-resolution (HR) images from their corresponding low-resolution (LR) images. In many remote sensing (RS) applications, spatial resolution of the aerial or satellite imagery has a great impact on the accuracy and reliability of information extracted from the images. In this study, the potential of a DCNN-based SISR model, called enhanced super-resolution generative adversarial network (ESRGAN), to predict the spatial information degraded or lost in a hyper-spatial resolution unmanned aircraft system (UAS) RGB image set is investigated. ESRGAN model is trained over a limited number of original HR (50 out of 450 total images) and virtually-generated LR UAS images by downsampling the original HR images using a bicubic kernel with a factor × 4 . Quantitative and qualitative assessments of super-resolved images using standard image quality measures (IQMs) confirm that the DCNN-based SISR approach can be successfully applied on LR UAS imagery for spatial resolution enhancement. The performance of DCNN-based SISR approach for the UAS image set closely approximates performances reported on standard SISR image sets with mean peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) index values of around 28 dB and 0.85 dB, respectively. Furthermore, by exploiting the rigorous Structure-from-Motion (SfM) photogrammetry procedure, an accurate task-based IQM for evaluating the quality of the super-resolved images is carried out. Results verify that the interior and exterior imaging geometry, which are extremely important for extracting highly accurate spatial information from UAS imagery in photogrammetric applications, can be accurately retrieved from a super-resolved image set. The number of corresponding keypoints and dense points generated from the SfM photogrammetry process are about 6 and 17 times more than those extracted from the corresponding LR image set, respectively.

Robust Prior-Based Single Image Super Resolution Under Multiple Gaussian Degradations

Although SISR (Single Image Super Resolution) problem can be effectively solved by deep learning based methods, the training phase often considers single degradation type such as bicubic interpolation or Gaussian blur with fixed variance. These priori hypotheses often fail and lead to reconstruction error in real scenario. In this paper, we propose an end-to-end CNN model RPSRMD to handle SR problem in multiple Gaussian degradations by extracting and using as side information a shared image prior that is consistent in different Gaussian degradations. The shared image prior is generated by an AED network RPGen with a rationally designed loss function that contains two parts: consistency loss and validity loss. These losses supervise the training of AED to guarantee that the image priors of one image with different Gaussian blurs to be very similar. Afterwards we carefully designed a SR network, which is termed as PResNet (Prior based Residual Network) in this paper, to efficiently use the image priors and generate high quality and robust SR images when unknown Gaussian blur is presented. When we applied variant Gaussian blurs to the low resolution images, the experiments prove that our proposed RPSRMD, which includes RPGen and PResNet as two core components, is superior to many state-of-the-art SR methods that were designed and trained to handle multi-degradation.

Exploiting Raw Images for Real-Scene Super-Resolution.

Super-resolution is a fundamental problem in computer vision which aims to overcome the spatial limitation of camera sensors. While significant progress has been made in single image super-resolution, most algorithms only perform well on synthetic data, which limits their applications in real scenarios. In this paper, we study the problem of real-scene single image super-resolution to bridge the gap between synthetic data and real captured images. We focus on two issues of existing super-resolution algorithms: lack of realistic training data and insufficient utilization of visual information obtained from cameras. To address the first issue, we propose a method to generate more realistic training data by mimicking the imaging process of digital cameras. For the second issue, we develop a two-branch convolutional neural network to exploit the radiance information originally-recorded in raw images. In addition, we propose a dense channel-attention block for better image restoration as well as a learning-based guided filter network for effective color correction. Our model is able to generalize to different cameras without deliberately training on images from specific camera types. Extensive experiments demonstrate that the proposed algorithm can recover fine details and clear structures, and achieve high-quality results for single image super-resolution in real scenes.

Benefiting from multitask learning to improve single image super-resolution

Despite significant progress toward super resolving more realistic images by deeper convolutional neural networks (CNNs), reconstructing fine and natural textures still remains a challenging problem. Recent works on single image super resolution (SISR) are mostly based on optimizing pixel and content wise similarity between recovered and high-resolution (HR) images and do not benefit from recognizability of semantic classes. In this paper, we introduce a novel approach using categorical information to tackle the SISR problem; we present an encoder architecture able to extract and use semantic information to super-resolve a given image by using multitask learning, simultaneously for image super-resolution and semantic segmentation. To explore categorical information during training, the proposed decoder only employs one shared deep network for two task-specific output layers. At run-time only layers resulting HR image are used and no segmentation label is required. Extensive perceptual experiments and a user study on images randomly selected from COCO-Stuff dataset demonstrate the effectiveness of our proposed method and it outperforms the state-of-the-art methods.